Papers with prompt tuning methods
Dual Context-Guided Continuous Prompt Tuning for Few-Shot Learning (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt-based paradigms have shown their competitive performance in many NLP tasks, but their effectiveness varies upon the model and training data. |
| Approach: | They propose a dual context-guided continuous prompt tuning method that integrates contextual information into the input input. |
| Outcome: | The proposed method outperforms existing prompt tuning methods in the few-shot setting and can be used in many NLP tasks. |
Prompt Tuning for Unified Multimodal Pretrained Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning has demonstrated success in natural language pretraining and even vision pretraining. |
| Approach: | They propose to apply prompt tuning to a unified sequence-to-sequence pretrained model by adding a sequence of learnable embeddings to each layer and finetuning the pretrained models on downstream tasks. |
| Outcome: | The proposed method outperforms other parameter-efficient tuning methods on multimodal models and is robust against adversarial attacks. |
CPL: Counterfactual Prompt Learning for Vision and Language Models (2022.emnlp-main)
Copied to clipboard
Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Wang
| Challenge: | Existing prompt tuning methods tend to learn spurious or entangled representations, leading to poor generalization to unseen concepts. |
| Approach: | They propose a prompt tuning technique that tunes the learnable prompt for pre-trained vision and language models. |
| Outcome: | The proposed method improves few-shot performance on vision and language tasks over existing prompt tuning methods. |
Generative Prompt Tuning for Relation Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing prompt tuning methods for RC are limited by label spaces and rigid prompt restrictions. |
| Approach: | They propose a generative prompt tuning method to reformulate relation classification as an infilling problem by adding cloze-style phrases to masked language modeling problems. |
| Outcome: | The proposed method exploits rich semantics of entity and relation types and can predict label verbalizations with varying lengths at multiple predicted positions. |
Towards Alleviating the Object Bias in Prompt Tuning-based Factual Knowledge Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to optimize prompts for factual knowledge extraction are undesirable object bias. |
| Approach: | They propose a prompt tuning method that reduces object bias and improves factual knowledge extraction. |
| Outcome: | The proposed method reduces object bias and improves accuracy of factual knowledge extraction. |
Continued Pretraining for Better Zero- and Few-Shot Promptability (2022.emnlp-main)
Copied to clipboard
Zhaofeng Wu, Robert L Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, Iz Beltagy
| Challenge: | Recent language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. |
| Approach: | They propose to use a dedicated pretraining stage to improve promptability in zero-shot settings and few-shot tuning. |
| Outcome: | The proposed method improves promptability in zero- and few-shot settings, while the existing method yields subpar performance. |
CodePrompt: Task-Agnostic Prefix Tuning for Program and Language Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-tuning methods have been used to solve inefficient parameter update and storage issues in Natural Language Generation tasks. |
| Approach: | They propose a task-agnostic prompt tuning method that reflects the traits of PLM for program language. |
| Outcome: | The proposed method is effective in three PLG tasks, not only in the full-data setting but also in the low-resource setting and cross-domain setting. |
IDPG: An Instance-Dependent Prompt Generation Method (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing prompt tuning methods use a fixed prompt in each input instance during the model training stage. |
| Approach: | They propose a conditional prompt generation method to generate prompts for each input instance. |
| Outcome: | The proposed method outperforms other prompt tuning methods while tuning fewer parameters. |
PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt tuning methods have training instability issues due to large variance of scores . existing prompt tuning algorithms have training stability issues due a slight change of input data . |
| Approach: | They propose an algorithm that smooths the loss landscape of vanilla prompt tuning by perturbation-based regularizers. |
| Outcome: | The proposed method improves the state-of-the-art prompt tuning methods by 1.94% and 2.34% on SuperGLUE and FewGLUE benchmarks. |